Image processing apparatus, image processing method, and program

ABSTRACT

A same region detector detects a same region of a processing target for each of a plurality of different viewpoints from polarization images in a plurality of polarization directions acquired for each of the viewpoints. The polarization images in the plurality of polarization directions acquired for each of the plurality of different viewpoints are, for example, polarization images acquired by imaging over a period of a plurality of frames in which a positional relationship between the processing target and a polarization image acquisition unit that acquires the polarization images changes. A polarization degree calculation unit calculates a polarization degree of the same region for each of the viewpoints on the basis of the polarization images in the plurality of polarization directions. A reflection removal unit performs reflection removal processing on the same region of the processing target by using the polarization images in the plurality of polarization directions of the viewpoint at which the polarization degree calculated by the polarization degree calculation unit is maximized. A reflection component can be removed even when an angle between a plane direction of a reflecting surface and an imaging direction is not clear.

TECHNICAL FIELD

This technology relates to an image processing apparatus, an imageprocessing method, and a program, and enables removal of a reflectioncomponent even when an angle formed by a plane direction of a reflectingsurface and an imaging direction is not clear.

BACKGROUND ART

Conventionally, not only a color image but also a polarization image issimultaneously acquired as disclosed in Patent Document 1, andcalculation of a polarization degree and a normal line, separation of adiffuse reflection component and a specular reflection component, andthe like are performed on the basis of the polarization image. Inaddition, Patent Document 2 discloses that a reflection angle, which isan angle formed by a plane direction corresponding to a normal directionof a reflecting surface and an imaging direction, is input so that areflection component can be removed even in a case where the reflectionangle does not form a Brewster's angle.

CITATION LIST Patent Document Patent Document 1: Japanese PatentApplication Laid-Open No. Patent Document 2: International PublicationNo. 2018/092540 SUMMARY OF THE INVENTION Problems to be Solved by theInvention

Meanwhile, in a case where a reflection component of a color image isremoved by using a polarization image, it is necessary to input areflection angle in Patent Document 2. It is thus not possible to copewith a reflecting surface different from the input reflection angle.

Therefore, an object of this technology is to provide an imageprocessing apparatus, an image processing method, and a program capableof removing a reflection component even when an angle formed by a planedirection of a reflecting surface and an imaging direction is not clear.

Solutions to Problems

A first aspect of this technology is an image processing apparatusincluding a same region detector that detects a same region of aprocessing target for each of a plurality of different viewpoints frompolarization images in a plurality of polarization directions acquiredfor each of the viewpoints, a polarization degree calculation unit thatcalculates a polarization degree of the same region for each of theviewpoints on the basis of the polarization images in the plurality ofpolarization directions, and a reflection removal unit that performsreflection removal processing on the same region of the processingtarget by using the polarization images in the plurality of polarizationdirections of the viewpoint at which the polarization degree calculatedby the polarization degree calculation unit is maximized.

In this technology, a same region detector detects a same region of aprocessing target for each of a plurality of different viewpoints frompolarization images in a plurality of polarization directions acquiredfor each of the viewpoints. The polarization images in the plurality ofpolarization directions acquired for each of the plurality of differentviewpoints are polarization images acquired by imaging over a period ofa plurality of frames in which a positional relationship between theprocessing target and the polarization image acquisition unit thatacquires the polarization images changes, for example, polarizationimages acquired by imaging the processing target that moves, orpolarization images acquired by performing imaging by the polarizationimage acquisition unit that moves. Furthermore, the polarization imagesin the plurality of polarization directions acquired for the pluralityof different viewpoints are polarization images acquired by providingpolarization image acquisition units that acquire the polarizationimages at a plurality of viewpoint positions and imaging the processingtarget by each of the polarization image acquisition units.

The polarization degree calculation unit calculates a polarizationdegree of the same region for each of the viewpoints on the basis of thepolarization images in the plurality of polarization directions, and thereflection removal unit performs the reflection removal processing onthe same region of the processing target by using the polarizationimages in the plurality of polarization directions of the viewpoint atwhich the polarization degree calculated by the polarization degreecalculation unit is maximized. Furthermore, in a case where a positionof the processing target is fixed, a position of the viewpoint at whichthe polarization degree calculated by the polarization degreecalculation unit is maximized may be stored together with the positionof the processing target, and the reflection removal unit may performthe reflection removal processing by using the polarization images ofthe processing target, the polarization images being acquired by thepolarization image acquisition unit at a position of the viewpoint atwhich the polarization degree is maximized.

A second aspect of this technology is an image processing methodincluding detecting, by a same region detector, a same region of aprocessing target for each of a plurality of different viewpoints frompolarization images in a plurality of polarization directions acquiredfor each of the viewpoints, calculating a polarization degree of thesame region by a polarization degree calculation unit for each of theviewpoints on the basis of the polarization images in the plurality ofpolarization directions, and performing reflection removal processing onthe same region of the processing target by a reflection removal unit byusing the polarization images in the plurality of polarizationdirections of the viewpoint at which the polarization degree calculatedby the polarization degree calculation unit is maximized.

A third aspect of this technology is a program that causes a computer toremove a reflection component of a processing target, the programcausing the computer to execute a procedure of detecting a same regionof the processing target for each of a plurality of different viewpointsfrom polarization images in a plurality of polarization directionsacquired for each of the viewpoints, a procedure of calculating apolarization degree of the same region for each of the viewpoints on thebasis of the polarization images in the plurality of polarizationdirections, and a procedure of performing reflection removal processingon the same region of the processing target by using the polarizationimages in the plurality of polarization directions of the viewpoint atwhich the polarization degree having been calculated is maximized.

Note that the program of the present technology is, for example, aprogram that can be provided to a general-purpose computer capable ofexecuting various program codes by a storage medium provided in acomputer-readable format, a communication medium, for example, a storagemedium such as an optical disk, a magnetic disk, a semiconductor memory,or the like, or a communication medium such as a network or the like. Byproviding such a program in a computer-readable format, processingaccording to the program is implemented on the computer.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram for describing P waves and S waves.

FIG. 2 is a diagram showing specular reflection of S waves and P waveswith respect to an incident angle.

FIG. 3 is a diagram for describing a luminance change of a polarizationimage.

FIG. 4 is a diagram exemplifying a relationship between a luminance anda polarization angle.

FIG. 5 is a diagram exemplifying a configuration of an embodiment.

FIG. 6 is a diagram exemplifying a configuration of a polarization imageacquisition unit.

FIG. 7 is a diagram exemplifying a pixel configuration in a plurality ofpolarization directions.

FIG. 8 is a flowchart exemplifying an operation according to theembodiment.

FIG. 9 is a diagram showing an operation example of the embodiment.

MODE FOR CARRYING OUT THE INVENTION

Hereinafter, an embodiment for implementing the present technology willbe described. Note that the description will be made in the followingorder.

1. Principle and problem of reflected light removal processing

2. Configuration of embodiment

3. Operation of embodiment

4. Application examples

<1. Principle and Problem of Reflected Light Removal Processing>

Next, a principle and problem of reflected light removal processing willbe described. FIG. 1 is a diagram for describing P waves and S waves.The P waves represent light having only a component parallel to anincident surface, and the S waves represent light having only acomponent perpendicular to the incident surface. FIG. 2 shows specularreflection of the S waves and the P waves with respect to an incidentangle. Note that a solid line indicates a characteristic of the S waves,and a broken line indicates a characteristic of the P waves.

It is known that reflectivity of P waves becomes substantially 0 whenattention is paid to an incident angle (=reflection angle) of about 70degrees, and an angle at which reflectivity of the P waves becomessubstantially “0” is called a Brewster's angle. Therefore, when theincident angle becomes the Brewster's angle, the reflected lightincludes no component of the P waves but only a component of the Swaves. Thus, the reflection component can be removed by removing thecomponent of the S waves by a polarizing filter. However, in a casewhere the reflection angle is an angle away from the Brewster's angle,the reflected light includes the component of the S waves and thecomponent of the P waves. It is therefore not possible to acquire animage from which the reflection component is removed by reducing thecomponent of the S waves by the polarizing filter.

Therefore, an image processing apparatus of the present disclosuresolves the problem of the angle between a plane direction of areflecting surface and an imaging direction, and performs processing ofremoving or reducing a reflected light component from not only acaptured image in a specific direction limited by the Brewster's anglebut also images captured from various directions to acquire a clearsubject image (transmitted light image or transmitted light componentimage). Specifically, a same region of a processing target is detectedfrom the polarization images in a plurality of polarization directionsacquired for a plurality of different viewpoints, a polarization degreeof the same region is calculated for each of the viewpoints, andreflection removal processing of the same region is performed by usingthe polarization images in the plurality of polarization directions of aviewpoint at which a polarization degree is maximized.

FIG. 3 is a diagram for describing a luminance change of a polarizationimage. As illustrated in FIG. 3 , a subject OB is illuminated by using alight source LT, and the subject OB is imaged by an imaging unit CM viaa polarizing plate PL. In this case, in the polarization image generatedby the imaging unit CM, it is known that a luminance of the subject OBchanges in accordance with rotation of the polarizing plate PL. Here,the highest luminance when the polarizing plate PL is rotated is definedas Imax, and the lowest luminance is defined as Imin. In addition, in acase where an x axis and a y axis in two-dimensional coordinates are aplanar direction of the polarizing plate PL, an angle on an xy planewith respect to the x axis when the polarizing plate PL is rotated isdefined as a polarization angle υpol. When the polarizing plate PL isrotated by 180 degrees, the polarizing plate PL returns to an originalpolarization state and has a cycle of 180 degrees.

FIG. 4 exemplifies a relationship between the luminance and thepolarization angle. The polarization angle υpol when the maximumluminance Imax is observed is defined as an azimuth angle φ. When such adefinition is made, a polarization model expression indicating a changein a luminance Ipol observed when the polarizing plate PL is rotated,that is, a predetermined luminance change caused by a difference in thepolarization angle can be represented as equation (1).

$\begin{matrix}\left\lbrack {{Equation}1} \right\rbrack &  \\{I_{pol} = {\frac{I_{\max} + I_{\min}}{2} + {\frac{I_{\max} - I_{\min}}{2}\cos\left( {\upsilon_{pol}\ —\ \phi} \right)}}} & (1)\end{matrix}$

In equation (1), the polarization angle υpol is clear upon generation ofa polarization image, and the maximum luminance Imax, the minimumluminance Imin, and the azimuth angle φ are variables. Thus, since thenumber of variables is three, fitting to a function shown in equation(1) can be performed by using, for example, the luminance of thepolarization image in three or more polarization directions. Inaddition, a polarization degree p can be calculated by performing acalculation of equation (2) using the maximum luminance Imax and theminimum luminance Imin. Furthermore, as is clear from equation (2), whena specular reflection component is large, the polarization degree p islarge.

$\begin{matrix}\left\lbrack {{Equation}2} \right\rbrack &  \\{\rho = \frac{I_{\max}I_{\min}}{I_{\max} + I_{\min}}} & (2)\end{matrix}$

In addition, specular reflection occurs due to a dominant light sourcein an imaging scene. Furthermore, since a correction value is adjustedin accordance with a color of illumination in a white balancecorrection, it is considered that the color of the illumination thatgenerates specular reflection is achromatic. In this case, RGB valuesrepresenting the color of the illumination have the same value, and thespecular reflection component can be removed by obtaining a colordifference. Therefore, in the removal of specular reflection, forexample, as disclosed in Patent Document “International Publication No.2016-136085”, calculation is performed from a polarization image onwhich white balance correction and the like have been performed by usinga pixel value of a red pixel, a pixel value of a green image, and apixel value of a blue pixel in a same polarizing pixel block, and apolarization image from which a specular reflection component has beenremoved is generated. Furthermore, in the removal of specularreflection, for example, a polarization image from which a specularreflection component by the light source is removed may be generated onthe assumption that the light source is white.

2. Configuration of Embodiment

Next, a configuration of the embodiment will be described. FIG. 5exemplifies the configuration of the embodiment. An image processingsystem 10 includes a polarization image acquisition unit 20 and an imageprocessor 30.

The polarization image acquisition unit 20 images a subject and acquiresa polarization image. FIG. 6 exemplifies a configuration of thepolarization image acquisition unit. For example, as illustrated in (a)of FIG. 6 , the polarization image acquisition unit 20 performs imagingby disposing a polarizing plate 202 having a pixel configuration in aplurality of polarization directions in an image sensor 201. Note that,(a) of FIG. 6 exemplifies a case where the polarizing plate 202, eachpixel of which is a pixel in any of four different types of polarizationdirections (polarization directions are indicated by arrows), isdisposed on a front surface of the image sensor 201. Furthermore, asillustrated in FIG. 6 (b), the polarization image acquisition unit 20may generate a plurality of polarization images in differentpolarization directions by using a configuration of a multi-lens array.For example, a plurality of (four in the drawing) lenses 203 is providedon the front surface of the image sensor 201, and each of the lenses 203forms an optical image of a subject on an imaging surface of the imagesensor 201. In addition, a polarizing plate 204 is provided on a frontsurface of each lens 203, polarization directions (polarizationdirections are indicated by arrows) of the polarizing plate 204 are madedifferent directions, and a polarization image including pixels in aplurality of polarization directions is generated. The polarizationimage acquisition unit 20 configured in this manner enables acquisitionof a polarization image including pixels in a plurality of polarizationdirections by one imaging. Furthermore, as illustrated in (c) of FIG. 6, polarizing plates 212-1 to 212-4 in different polarization directionsmay be provided in front of imaging units 210-1 to 210-4 to generate aplurality of polarization images in different polarization directions.

Note that, in a case where movement of the subject is slow or in a casewhere the subject moves stepwise, as illustrated in (d) of FIG. 6 , apolarizing plate 211 may be provided in front of an imaging unit 210. Inthis case, the polarizing plate 211 is rotated to perform imaging in aplurality of different polarization directions, to acquire a pluralityof polarization images in different polarization directions.

The plurality of different polarization directions may be anycombination of angles as long as all the angles are different. Forexample, 0 degrees, 60 degrees, and 120 degrees are used in a case wherethree polarization directions are used, and 0 degrees, 45 degrees, 90degrees, and 135 degrees are used in a case where four polarizationdirections are used.

In a case where no color filter is used in the image sensor 201, thepolarization image acquisition unit 20 can acquire a luminancepolarization image. Here, in a case of (a) of FIG. 6 , it is possible toacquire an image equivalent to an unpolarized normal luminance image byaveraging luminance of four adjacent pixels in different polarizationdirections. In addition, in cases of (b) and (c) of FIG. 6 , when apositional interval of each lens 203 and the imaging units 210-1 to210-4 is negligibly short with respect to a distance to an abnormalitydetection target, disparity can be ignored in a plurality ofpolarization images in different polarization directions. It istherefore possible to acquire an image equivalent to the unpolarizednormal luminance image by averaging luminance of polarization images indifferent polarization directions. Furthermore, in a case where thedisparity cannot be ignored, it is possible to acquire an imageequivalent to the unpolarized normal luminance image by aligningpolarization images in different polarization directions in accordancewith a disparity amount and averaging a luminance of the alignedpolarization images. In addition, in a case of (d) of FIG. 6 , it ispossible to acquire an image equivalent to the unpolarized normalluminance image by averaging luminance of luminance polarization imagesin different polarization directions for each pixel.

Moreover, the polarization image acquisition unit 20 may simultaneouslygenerate not only the luminance polarization image but also threeprimary color images by providing a color filter in the image sensor201, or may simultaneously generate an infrared image or the like.Furthermore, the polarization image acquisition unit 20 may calculate aluminance from the three primary color images to generate a luminanceimage.

FIG. 7 exemplifies a pixel configuration in a plurality of polarizationdirections, and the configuration illustrated in FIG. 7 is repeated in ahorizontal direction and a vertical direction. Each of (a) and (b) ofFIG. 7 illustrates a pixel configuration in a case where ablack-and-white image is acquired. Note that (a) of FIG. 7 exemplifies acase where a polarizing pixel block of 2×2 pixels includes, for example,polarizing pixels in polarization directions (polarization angles) of 0degrees, 45 degrees, 90 degrees, and 135 degrees. In addition, (b) ofFIG. 7 exemplifies a case where a polarizing pixel block of 4×4 pixelsincludes, for example, polarizing pixels in polarization directions of 0degrees, 45 degrees, 90 degrees, and 135 degrees with 2×2 pixels as aunit of polarization direction. Note that in a case where a polarizationcomponent unit of the polarizing plate is 2×2 pixels as illustrated in(b) of FIG. 7 , a ratio of polarization component leakage from adjacentregions of different polarization component units to polarizationcomponents acquired for each polarization component unit is lower thanin a case of 1×1 pixel illustrated in (a) of FIG. 7 . In addition, in acase where a wire grid is used as the polarizing plate, polarized lighthaving an electric field component perpendicular to a direction of agrid (wire direction) is transmitted, and then the longer the wire, thehigher the transmittance. Thus, in a case where the polarizationcomponent unit is 2×2 pixels, the transmittance is higher than in thecase of 1×1 pixels. Therefore, in a case where the polarizationcomponent unit is 2×2 pixels, the transmittance is higher than in thecase of 1×1 pixels, and an extinction ratio can be improved.

Each of (c) to (g) of FIG. 7 exemplifies a pixel configuration in a casewhere a color image is acquired. (c) of FIG. 7 illustrates a case wherethe polarizing pixel block of 2×2 pixels illustrated in (a) of FIG. 7 isset as one color unit, and three primary color pixels (a red pixel, agreen pixel, and a red pixel) are arranged in a Bayer array.

(d) of FIG. 7 exemplifies a case where the three primary color pixelsare provided in the Bayer array for each pixel block of 2×2 pixels in asame polarization direction illustrated in (b) of FIG. 7 .

(e) of FIG. 7 exemplifies a case where the three primary color pixelsare provided in the Bayer array for each pixel block of 2×2 pixels inthe same polarization direction, and blocks of 2×2 pixels in differentpolarization directions are set as pixels of a same color.

(f) of FIG. 7 illustrates a case where, for a pixel block of 2×2 pixelsin the Bayer array in the same polarization direction, a phasedifference in the polarization direction from a pixel block adjacent inthe horizontal direction is 90 degrees, and a phase difference in thepolarization direction from a pixel block adjacent in the verticaldirection is ±45 degrees.

(g) of FIG. 7 illustrates a case where, for a pixel block of 2×2 pixelsin the Bayer array in the same polarization direction, a phasedifference in the polarization direction from a pixel block adjacent inthe vertical direction is 90 degrees, and a phase difference in thepolarization direction from a pixel block adjacent in the horizontaldirection is ±45 degrees.

In addition, in the pixel configuration, a block of 2×2 pixels mayinclude the three primary color pixels and white pixels, or may includea pixel block of three primary colors and a pixel block of white.Furthermore, the block of 2×2 pixels may include polarizing pixels andnon-polarizing pixels in different polarization directions, or mayinclude a polarizing pixel block and a non-polarizing pixel block indifferent polarization directions. In a case where such a pixelconfiguration is used, the polarization image acquisition unit 20generates, for each polarization direction, polarization imaging pixelshaving a resolution in a pixel unit, that is, in units of pixel blocksof a predetermined number of pixels. Furthermore, in a case where thepixel block is in the Bayer array, polarization imaging pixels having aresolution in units of one pixel may be generated for each polarizationdirection by using existing demosaic processing.

The polarization image acquisition unit 20 acquires polarization imagesin a plurality of polarization directions for each of a plurality ofdifferent viewpoints. For example, as the configuration of (a) of FIG. 2, (b) of FIG. 2 , or (c) of FIG. 2 , the polarization image acquisitionunit 20 images a moving subject over a period of a plurality of framesto acquire polarization images in a plurality of polarization directionsfor each of a plurality of different viewpoints. Furthermore, thepolarization image acquisition unit 20 may be provided at a plurality ofdifferent viewpoint positions to image the subject, and then each of theplurality of polarization image acquisition units 20 may acquirepolarization images in a plurality of polarization directions.

Returning to FIG. 5 , the image processor 30 includes a same regiondetector 31, a polarization degree calculation unit 32, a storage 33, animage selection unit 34, and a reflection removal unit 35.

The same region detector 31 detects the same region of the processingtarget for each viewpoint from the polarization image acquired by thepolarization image acquisition unit 20. For example, the same regiondetector 31 detects the same region of the processing target by usingthe polarization image acquired by the polarization image acquisitionunit 20 and the polarization image stored in the storage 33 as describedlater. The same region detector 31 outputs a same region detectionresult to the image selection unit 34. Note that, when the same regiondetector 31 detects the same region of the processing target from eachpolarization image, if a shape change in the polarization image of thesubject caused by movement of the subject or a difference in viewpointposition of the polarization image acquisition unit 20 is corrected, thesame region can be easily detected by using the corrected polarizationimage.

The polarization degree calculation unit 32 calculates a polarizationdegree for each pixel unit of the polarization image for each of aplurality of different viewpoints. The polarization degree calculationunit 32 calculates a polarization degree for each pixel unit on thebasis of equation (2) by using polarization images in a plurality ofpolarization directions. In addition, the polarization degreecalculation unit 32 calculates the polarization degree for each of theplurality of different viewpoints, and outputs the calculatedpolarization degree to the storage 33.

The storage 33 stores the polarization images in the plurality ofpolarization directions acquired for each of the plurality of differentviewpoints by the polarization image acquisition unit 20. In addition,the storage 33 stores the polarization degree calculated for each pixelunit for each of the plurality of viewpoints.

The image selection unit 34 reads the polarization degree of the sameregion detected by the same region detector 31 from the storage 33 foreach of the plurality of viewpoints. In addition, the image selectionunit 34 compares the read polarization degrees to determine a viewpointat which the polarization degree is maximized. For example, in a casewhere the same region is a plurality of pixel regions, the imageselection unit 34 determines the viewpoint at which the polarizationdegree is maximized for each pixel position. Furthermore, the imageselection unit 34 reads the polarization images for each of theplurality of polarization directions acquired from the viewpoint atwhich the polarization degree is maximized from the storage 33 andoutputs the polarization images to the reflection removal unit 35.

The reflection removal unit 35 removes the specular reflection componentby using the polarization image selected by the image selection unit 34.Note that removal of the specular reflection component is only requiredto be performed by using a method disclosed in, for example, PatentDocument “International Publication No. 2016/136085”, Patent Document“International Publication No. 2018/230119”, or the like.

3. Operation of Embodiment

FIG. 8 is a flowchart exemplifying an operation according to theembodiment. In step ST1, the image processor acquires a polarizationimage. The image processor 30 acquires polarization images in aplurality of polarization directions for each of a plurality ofdifferent viewpoints from the polarization image acquisition unit 20,and the processing proceeds to step ST2.

In step ST2, the image processor detects the same region. The sameregion detector 31 of the image processor 30 detects the same region ofthe processing target for each viewpoint from the polarization imageacquired in step ST1, and the processing proceeds to step ST3.

In step ST3, the image processor calculates a polarization degree. Thepolarization degree calculation unit 32 of the image processor 30calculates the polarization degree for each viewpoint for each pixelunit by using the polarization images in the plurality of polarizationdirections, and the processing proceeds to step ST4.

In step ST4, the image processor selects a polarization image. The imageselection unit 34 of the image processor 30 determines the viewpoint atwhich the polarization degree calculated in step ST3 is maximized foreach pixel position in the same region detected in step ST2, and selectsthe polarization image of the viewpoint at which the polarization degreeis maximized, and the processing proceeds to step ST5.

In step ST5, the image processor performs reflection removal processing.The reflection removal unit 35 of the image processor 30 removes thespecular reflection component by using the polarization image selectedin step ST4, and sets an image of the same region detected in step ST2as an image from which reflection is removed or suppressed.

FIG. 9 illustrates an operation example of the embodiment. A processingtarget is a moving vehicle OBm. The polarization image acquisition unit20 is fixed ahead in front of the vehicle OBm, and images the vehicleOBm to acquire a polarization image. Note that a normal line of theregion FP on the windshield of the vehicle OBm is a normal line SNfp. Inaddition, the image processor 30 (not shown) performs the reflectionremoval processing on the region FP in the windshield of the vehicle OBmby using the polarization image acquired by imaging over a period of aplurality of frames in which a positional relationship between, forexample, the vehicle OBm as the processing target and the polarizationimage acquisition unit 20 that acquires the polarization image changes,as the polarization images in the plurality of polarization directionsacquired for each of the plurality of different viewpoints.

(a) of FIG. 9 exemplifies a case where the vehicle OBm at time point t1is at a position P1. In this case, an emission light from the region FPincident on the polarization image acquisition unit 20 has a reflectionangle α1. (b) of FIG. 9 exemplifies a case where the vehicle OBm at timepoint t2 is at a position P2. In this case, an emission light from theregion FP incident on the polarization image acquisition unit 20 has areflection angle α2. (c) of FIG. 9 exemplifies a case where the vehicleOBm at time point t3 is at the position P1. In this case, an emissionlight from the region FP incident on the polarization image acquisitionunit 20 has a reflection angle α3.

The image processor 30 calculates the polarization degree from thepolarization images acquired by the polarization image acquisition unit20 at each time point on the basis of the polarization images in theplurality of polarization directions in the region FP. Furthermore, theimage processor 30 selects the polarization image at which thepolarization degree is maximized. For example, when the reflection angleα2 is the Brewster's angle, the degree of polarization is maximized.Therefore, the image processor 30 uses polarization images in aplurality of polarization directions indicating the region FP acquiredat time point t2 at which the polarization degree is maximized. In thiscase, since there are few P waves in the emission light from the regionFP at time point t2, it is possible to obtain a captured image in whichthe specular reflection is suppressed by performing the reflectionremoval processing by using the polarization images in the plurality ofpolarization directions indicating the region FP acquired at time pointt2 without setting the angle and the like of the emission light.

Furthermore, FIG. 9 exemplifies a case where the processing target ismoving, but the position of the processing target may be fixed, and thepolarization image acquisition unit may move. In addition, in a casewhere the position of the processing target is fixed and thepolarization image acquisition unit moves, the position of the viewpointat which the polarization degree calculated by the polarization degreecalculation unit is maximized may be stored together with the positionof the processing target. As described above, by storing the position ofthe processing target and the position of the viewpoint at which thepolarization degree is maximized, in a case where the polarization imageacquisition unit is provided in each moving body, the reflectioncomponent can be easily removed by using the polarization image acquiredat the position based on stored information without calculating thepolarization degree or determining the position of the viewpoint atwhich the calculated polarization degree is maximized in each movingbody. For example, in a case where a building or the like that generatesreflection is indicated by map information, a polarization image isacquired at a position of a viewpoint stored in association with aposition of the building that generates reflection, and reflectiongenerated by the building or the like can be easily removed.

4. Application Examples

The technology of the present disclosure can be applied to variousfields. For example, the technology of the present disclosure may beimplemented as a device mounted on any type of mobile body such as anautomobile, an electric vehicle, a hybrid electric vehicle, amotorcycle, a bicycle, a personal mobility, an airplane, a drone, aship, a robot, and the like. In addition, it may be implemented as adevice mounted on equipment used in a production process in a factory orequipment used in a construction field. In a case where the imageprocessing apparatus of the present technology is applied to such afield, the reflection component can be easily removed, and thus, asystem that enables safer traveling and the like can be constructed. Inaddition, in an application to a monitoring field or the like, areflection component of a moving monitoring target can be easilyremoved.

The series of processing described herein can be executed by hardware,software, or a combined configuration of both. In a case of executingprocessing by software, a program storing a processing sequence isinstalled in a memory in a computer incorporated in dedicated hardwareand executed. Alternatively, the program can be installed and executedin a general-purpose computer capable of executing various types ofprocessing.

For example, the program can be recorded in advance in a hard disk, asolid state drive (SSD), or a read only memory (ROM) as a recordingmedium. Alternatively, the program can be temporarily or permanentlystored (recorded) in a removable recording medium such as a flexibledisk, a compact disc read only memory (CD-ROM), a magneto optical (MO)disk, a digital versatile disc (DVD), a Blu-ray disc (BD) (registeredtrademark), a magnetic disk, a semiconductor memory card, or the like.Such a removable recording medium can be provided as so-called packagesoftware.

Furthermore, in addition to installing the program from the removablerecording medium to the computer, the program may be transferred from adownload site to the computer wirelessly or by wire via a network suchas a local area network (LAN), the Internet, or the like. In thecomputer, the program thus transferred can be received and installed ina recording medium such as a built-in hard disk or the like.

Note that the effects herein described are merely examples and are notlimited, and furthermore, additional effects that are not describedherein may be obtained. Furthermore, the present technology should notbe construed as being limited to the embodiment of the technologydescribed above. The embodiment of this technology discloses the presenttechnology in the form of exemplification, and it is obvious that thoseskilled in the art can make modifications or substitutions of theembodiment without departing from the gist of the present technology.That is, in order to determine the gist of the present technology, theclaims should be taken into consideration.

Furthermore, the image processing apparatus of the present technologycan adopt the following configurations.

(1) An image processing apparatus includes a same region detector thatdetects a same region of a processing target for each of a plurality ofdifferent viewpoints from polarization images in a plurality ofpolarization directions acquired for each of the viewpoints, apolarization degree calculation unit that calculates a polarizationdegree of the same region for each of the viewpoints on the basis of thepolarization images in the plurality of polarization directions, and areflection removal unit that performs reflection removal processing onthe same region of the processing target by using the polarizationimages in the plurality of polarization directions of the viewpoint atwhich the polarization degree calculated by the polarization degreecalculation unit is maximized.

(2) In the image processing apparatus according to (1), the polarizationimages in the plurality of polarization directions acquired for each ofthe plurality of different viewpoints are polarization images acquiredby imaging over a period of a plurality of frames in which a positionalrelationship between the processing target and the polarization imageacquisition unit that acquires the polarization images changes.

(3) In the image processing apparatus according to (2), the polarizationimages are polarization images acquired by imaging the processing targetthat moves.

(4) In the image processing apparatus according to (2), the polarizationimages are polarization images acquired by performing imaging by thepolarization image acquisition unit that moves.

(5) In the image processing apparatus according to (1), the polarizationimages in the plurality of polarization directions acquired for theplurality of different viewpoints are polarization images acquired byproviding polarization image acquisition units that acquire thepolarization images at a plurality of viewpoint positions and imagingthe processing target by each of the polarization image acquisitionunits.

(6) In the image processing apparatus according to any of (1) to (5),the polarization degree calculation unit calculates the polarizationdegree for each pixel unit.

(7) In the image processing apparatus according to any of (1) to (6), ina case where a position of the processing target is fixed, a position ofthe viewpoint at which the polarization degree calculated by thepolarization degree calculation unit is maximized is stored togetherwith the position of the processing target.

(8) In the image processing apparatus according to (7), the reflectionremoval unit performs the reflection removal processing by using thepolarization images of the processing target, the polarization imagesbeing acquired by the polarization image acquisition unit at theposition of the viewpoint at which the polarization degree is maximized.

REFERENCE SIGNS LIST

-   10 Image processing system-   20 Polarization image acquisition unit-   30 Image processor-   31 Same region detector-   32 Polarization degree calculation unit-   33 Storage-   34 Image selection unit-   35 Reflection removal unit-   201 Image sensor-   202, 204, 211, 212-1 to 212-4 Polarizing plate-   203 Lens-   210, 210-1 to 210-4 Imaging unit

1. An image processing apparatus comprising: a same region detector thatdetects a same region of a processing target for each of a plurality ofdifferent viewpoints from polarization images in a plurality ofpolarization directions acquired for each of the viewpoints; apolarization degree calculation unit that calculates a polarizationdegree of the same region for each of the viewpoints on a basis of thepolarization images in the plurality of polarization directions; and areflection removal unit that performs reflection removal processing onthe same region of the processing target by using the polarizationimages in the plurality of polarization directions of the viewpoint atwhich the polarization degree calculated by the polarization degreecalculation unit is maximized.
 2. The image processing apparatusaccording to claim 1, wherein the polarization images in the pluralityof polarization directions acquired for each of the plurality ofdifferent viewpoints are polarization images acquired by imaging over aperiod of a plurality of frames in which a positional relationshipbetween the processing target and the polarization image acquisitionunit that acquires the polarization images changes.
 3. The imageprocessing apparatus according to claim 2, wherein the polarizationimages are polarization images acquired by imaging the processing targetthat moves.
 4. The image processing apparatus according to claim 2,wherein the polarization images are polarization images acquired byperforming imaging by the polarization image acquisition unit thatmoves.
 5. The image processing apparatus according to claim 1, whereinthe polarization images in the plurality of polarization directionsacquired for the plurality of different viewpoints are polarizationimages acquired by providing polarization image acquisition units thatacquire the polarization images at a plurality of viewpoint positionsand imaging the processing target by each of the polarization imageacquisition units.
 6. The image processing apparatus according to claim1, wherein the polarization degree calculation unit calculates thepolarization degree for each pixel unit.
 7. The image processingapparatus according to claim 1, wherein in a case where a position ofthe processing target is fixed, a position of the viewpoint at which thepolarization degree calculated by the polarization degree calculationunit is maximized is stored together with the position of the processingtarget.
 8. The image processing apparatus according to claim 7, whereinthe reflection removal unit performs the reflection removal processingby using the polarization images of the processing target, thepolarization images being acquired by the polarization image acquisitionunit at the position of the viewpoint at which the polarization degreeis maximized.
 9. An image processing method comprising: detecting, by asame region detector, a same region of a processing target for each of aplurality of different viewpoints from polarization images in aplurality of polarization directions acquired for each of theviewpoints; calculating a polarization degree of the same region by apolarization degree calculation unit for each of the viewpoints on abasis of the polarization images in the plurality of polarizationdirections; and performing reflection removal processing on the sameregion of the processing target by a reflection removal unit by usingthe polarization images in the plurality of polarization directions ofthe viewpoint at which the polarization degree calculated by thepolarization degree calculation unit is maximized.
 10. A program thatcauses a computer to remove a reflection component of a processingtarget, the program causing the computer to execute: a procedure ofdetecting a same region of the processing target for each of a pluralityof different viewpoints from polarization images in a plurality ofpolarization directions acquired for each of the viewpoints; a procedureof calculating a polarization degree of the same region for each of theviewpoints on a basis of the polarization images in the plurality ofpolarization directions; and a procedure of performing reflectionremoval processing on the same region of the processing target by usingthe polarization images in the plurality of polarization directions ofthe viewpoint at which the polarization degree having been calculated ismaximized.